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The  Technology  Readiness  Challenge 


When  the  nation's  first  ballistic  missile  rose  about  6  inches 
above  the  launch  pad  before  toppling  over  and  exploding, 
<Simon>  Ramo  reportedly  turned  to  an  Air  Force  general  and 
said:  "Well,  Benny,  now  that  we  know  the  thing  can  fly,  all  we 

have  to  do  is  improve  its  range  a  bit.” 

- Book  Review,  LA  Times,  July  5,  2009,  B4  Business,  by  Peter  Pae 


(Simon  Ramo  was  the  co-founder  of  and  the  "R"  in  TRW  Corporation,  now  part  of  Northrop 
Grumman  Corporation;  TRW  was  acquired  together  with  Litton,  Westinghouse,  Logicon,  etc.) 
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Outline 


•  Motivation 

•  Technology  Readiness  Assessments  -  the  64,000-foot  View 

•  Technology  Readiness  Assessments  -  the  10,000-foot  View 

•  Software  Technology  Readiness  Definitions 

•  Emerging  Software  Technologies  to  Watch 

•  Algorithms 

•  Environmental  Considerations 

•  What  Can  We  Learn  from  the  Software  Architecture? 

•  Selected,  High-Level  Viewpoints  for  Technology  Readiness  Assessment 

•  Critical  Technology  Element  Identification  Questions 

•  Software  Technology  Readiness  Determination 

•  Case  Studies 

-  Space  Vehicle:  Software  Critical  Technology  Element  Identification  for  a  Space  Vehicle 

-  Ground  System:  Is  Service-Oriented  Architecture  (SO A)  a  Critical  Technology  Element? 

•  Challenging  Topics 

•  Concluding  Thoughts 

•  Acronyms 

•  References 

•  Backup  Slides 
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Motivation 


•  Why  is  Technology  Readiness  Assessment  important? 

-  “The  inability  to  define  and  thus  measure  technology  readiness  facilitates 
decisions  to  incorporate  immature  technology  in  system  design  at 
Milestone  B  which  consequently  leads  to  technical  problems  during 
System  Design  and  Development” [DAPA  2006] 

•  Why  should  it  be  important  for  You? 

-  For  one  thing,  it  is  the  Law  (more  on  this  later.)  Nevertheless  ... 

-  If  you  are  in  the  Acquisition  Program  Office  (APO): 

•  You  might  have  to  provide  data  to  an  Independent  Review  Team  (IRT) 
conducting  a  Technology  Readiness  Assessment  (TRA) 

-  If  you  are  a  Contractor: 

•  You  might  want  to  gain  insight  into  how  your  proposals  are  evaluated 

-  If  you  are  in  The  Aerospace  Corporation: 

•  You  might  be  invited  to  become  a  member  of  an  IRT 

•  What  is  my  objective  in  preparing  this  course? 

-  Guidance  on  this  topic  will  always  be  inadequate  due  to  the  disruptive 
nature  of  technology  innovation.  I  want  to  encourage  you  to  understand  the 
underlying,  core  principles  rather  than  Just  trying  to  follow  the  letter  of  the 
DOD  Deskbook. 
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Technology  Readiness  Assessments  - 
the  64,000-foot  View 


•  Public  Law  109-163-Jan.6,  2006,  Section  801 

TITLE  VIII— ACQUISITION  POLICY,  ACQUISITION  MANAGEMENT,  AND 
RELATED  MATTERS 

Subtitle  A — Provisions  Relating  to  Major  Defense  Acquisition  Programs 
SEC.  801.  REQUIREMENT  FOR  CERTIFICATION  BEFORE  MAJOR 
DEFENSE  A  CQUISITION PROGRAM MA  Y PROCEED  TO  MILESTONE  B. 

(a)  CERTIFICATION  REQUIREMENT. — Chapter  139  of  title  10,  United  States  Code,  is 
amended  by  inserting  after  section  2366  the  following  new  section: 

**§  2366a.  Major  defense  acquisition  programs:  certification  required  before  Milestone  B 
or  Key  Decision  Point  B  approval 

(a)  CERTIFICATION. — A  major  defense  acquisition  program  may  not  receive  Milestone  B 
approval,  or  Key  Decision  Point  B  approval  in  the  case  of  a  space  program,  until  the 
milestone  decision  authority  certifies  that — 

(1)  the  technology  in  the  program  has  been  demonstrated  in  a  relevant  environment; ...” 


Note  that  the  term  “Key  Decision  Point”  is  not  in  use  since  the  cancellation  of  NSSAP  03-01 
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Technology  Readiness  Assessments  - 
the  64,000-foot  View  (Cont.) 

•  November  2,  2007  Air  Force  Memorandum  on  Technology  Certification 

-  Spells  out  that  for  all  Critical  Technology  Elements  (CTEs)  it  has  to  be 
demonstrated  in  a  relevant  environment  that  they  are  at  Technology 
Readiness  Level  (TRL)  6  or  greater. 

•  New  provisions  in  the  Weapon  Systems  Acquisition  Reform  Act  of  2009 

TITLE  I— ACQUISITION  ORGANIZATION 

SEC.  104.  Assessment  of  technological  maturity  of  critical  technologies  of  major  defense 
acquisition  programs  by  the  Director  of  Defense  Research  and  Engineering. 

(a)  ASSESSMENT  BY  DIRECTOR  OF  DEFENSE  RESEARCH  AND  ENGINEERING.— 

(1)  IN  GENERAL.  —  Section  139a  of  title  10,  United  States  Code,  is  amended  by  adding  at 
the  end  the  following  new  subsection: 

(c)  (1)  The  Director  of  Defense  Research  and  Engineering,  in  consultation  with  the 
Director  of  Developmental  Test  and  Evaluation,  shall  periodically  review  and  assess  the 
technological  maturity  and  integration  risk  of  critical  technologies  of  the  major  defense 
acquisition  programs  ... 

(2)  The  Director  shall  submit  to  the  Secretary  of  Defense  and  the  congressional 
defense  committees  by  March  1  of  each  year  a  report... 
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Basic  Department  of  Defense  (DOD)  TRA  Definitions 


•  The  key  document  providing  DOD  guidance  on  carrying  out  a  TRA  is 
entitled  the  “Technology  Readiness  Assessment  (TRA)  Deskbook” 

-  This  tutorial  is  based  on  the  most  recent,  July  2009  edition 

•  Technology  Maturity 

-  A  measure  or  degree  to  which  proposed  technologies  meet  program 
objectives 

•  Technology  Readiness  Assessment 

-  A  TRA  is  a  formal,  systematic,  metrics-based  process  and  accompanying 
report  that  assesses  the  maturity  of  criticai  hardware  and  software 
technologies  to  be  used  in  systems.  The  TRA  is  not  intended  to  predict 
future  performance  of  the  evaiuated  technoiogies,  nor  does  it  assess  the 
quality  of  the  system  architecture,  design,  or  integration  plan 

•  TRA  is  different  from  “Conventional”  Risk  Management 

-  The  resuit  of  a  TRA  is  a  singie  number  on  a  1-9,  ordinai  scale,  called 
Technology  Readiness  Level  (TRL). 

-  TRLs  do  not  intend  to  reflect  either  the  likelihood  of  attaining  required 
maturity  or  the  impact  of  not  achieving  the  required  maturity 

•  The  TRA  complements  -  but  does  not  in  any  way  preclude  -  the 
Program  Manager’s  responsibility  to  pursue  reduction  of  all  risks 
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Critical  Technology  Elements 


•  Context  for  Technology  Readiness  Assessments 

-  For  practical  purposes  not  all  planned  technologies  are  assessed 

•  The  technologies  that  are  subject  of  a  TRA  will  be  called  Critical 
Technology  Elements  (CTEs) 

-  However,  the  analysis  of  candidate  technologies  begins  even  before 
Materiel  Development  Decision  takes  place  for  the  acquisition 

•  A  technology  element  is  critical  if 

•  The  system  being  acquired  depends  on  this  technology  element  to  meet 
operational  requirements  within  acceptable  cost  and  schedule  limits,  and 

•  The  technology  element  or  its  application  is 

-  either  new  or  novel,  or 

-  in  an  area  that  poses  major  technological  risk  during  detailed  design 
or  demonstration 

-  Candidate  CTEs  vs.  CTEs 

•  Until  it  is  approved  by  the  Milestone  Decision  Authority  (MDA,)  all  CTEs 
are  considered  only  as  Candidate  CTEs 

-  CTE  identification  data  includes  the  criteria  and  rationale  for  declaring  the 
CTE  as  critical  or  eliminating  it  as  a  CTE  candidate 
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Technology  Evaluation  Logistics 


Pre-Systems  Acquisition 


Technology 

De\/elopment 

Approval 


Milestones: 


Source  Selection 


Engineering  and 
Manufacturing 
Development 
Approval 


DOD  5000.02  (2  December  2008) 
Systems  Acquisition 


Post-CDR 

Assessment 


Low-Rate 
Initial  Prod 
Approval 


IOC 


(Event-Based  but  Infonnal) 


(Event-Based,  Fonnal) 


(Event-Based,  Fomrial) 


Steps  of  a  formal,  IRT-conducted  TRA  at  Milestones  B  and  C 

•  The  Component  Science  &  Technology  Executive  appoints  an  IRT  of 
appropriate  Subject  Matter  Experts  (SMEs) 

•  Acquisition  Program  Office  presents  its  technology  plans  to  the  IRT 

•  IRT  evaluates  the  plan  and  submits  the  list  of  selected  CTEs  to  the 
Defense  Acquisition  Board  (DAB)  for  approval 

•  IRT  assesses  the  maturity  of  the  approved  CTEs 

•  IRT  briefs  the  Milestone  Decision  Authority  (MDA)  on  its  findings 

•  MDA  approves/disapproves  the  entry  to  the  next  acquisition  phase 
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Technology  Readiness  Assessments  - 
the  10,000-foot  View 


- 


Relevant  Environment  Definition 


-  Relevant  Environment  is  a  validation  environment  that  simulates  key  aspects 
of  the  Operational  Environment 

-  The  purpose  of  using  a  relevant  environment  is  to  demonstrate  sufficient 
confidence  in  the  CTE;  i.e.,  that  skillful  application  of  this  technology  will  fully 
support  the  required  threshold  functionality. 

•  Relevant  Environment  for  Space* 

-  A  satellite  from  launch  to  standard  operation  in  space  is  exposed  to 
drastically  changing  environmental  conditions  and  a  reievant  environment 
test  design  must  encompass  all  such  stressing,  aggregate  conditions: 

•  Space  Environment 

•  Launch  Environment 

•  Designed  Environment  This  is  where  software  “lives” 

•  Operational  Environment 

*This  is  an  experience-based  recommendation;  unfortunately  the  DOD  Deskbook  does  not 
have  adequate  space-related  guidance.  For  further  details  please  see  the  backup  slides. 
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Assessing  CTEs  using  the  TRL  “Thermometer” 


TRL  9 


Actual  system  proven  through  successful  mission  operations 


TRL  8 

TRL  7 


Actual  system  completed  and  qualified  through  test  and 
demonstration 

System  prototype  demonstration  in  an  operational  environment 


TRL  6 

TRL  5 


System/subsystem  model  or  prototype  demonstration  in  a  relevant 
environment 

Component  and/or  breadboard  validation  in  relevant  environment 


TRL  4 


Component  and/or  breadboard  validation  in  laboratory  environment 


TRL  3 


Analytical  and  experimental  critical  function  and/or  characteristic 
proof  of  concept 


TRL  2 


rRLi 

r 

Technology  concept  and/or  application  formulated 
Basic  principles  observed  and  reported 
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Before  We  Move  On... 


•  Check  your  understanding  of  the  following  new  terms 


-  IRT 

-  TRA 

-  TRL 

-  CTE 


-  Relevant  Environment 


•  O.K.? 
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Software  Technology  Readiness  Definitions* 

•  The  Definition  of  Software  Technology 

-  Software  technology  is  defined  as  the  theory  and  practice  of  various 
sciences  appiied  to  software  development,  operation,  understanding,  and 
maintenance.  Software  Technology  is  any  concept  process,  method, 
algorithm,  or  tool  whose  primary  purpose  is  the  development,  operation, 
understanding,  and  maintenance  of  software  [Foreman  1997] 

-  Software  technology  examples 

•  Technology  directly  used  in  the  objective  system 

-  E.g.,  two-tier  and  three-tier  architecture,  Service-Oriented 
Architecture  (SOA,)  Remote  Procedure  Calls  (RPC) 

•  Technology  used  in  tools  that  produce  or  maintain  software 

-  E.g.,  Graphical  User  Interface  (GUI)  builders,  programming 
languages  and  compilers,  cyclomatic  complexity  analyzers 

•  Process  technologies  applied  to  produce  or  maintain  software 

-  Personal  Software  Process  (PSP),  Clean  room  Software  Engineering. 

*  Unfortunately,  the  TRA  Deskbook  does  not  have  a  definition  of  software  technology 

0 AEROSPACE 


15 


A  Sampler  of  Emerging  Software  Technologies  to  Watch 


•  Technologies  directly  supporting  mission  requirements 

-  Network  Centric  Warfare  (NCW) 

•  NCW  is  a  state-of-the  art  war-fighting  theory  with  the  following,  two 
implementation  dimensions 

-  Network  Centric  Operations  (NCO),  deaiing  with  the  cognitive  and 
social  dimensions  of  NCW 

-  Network  Centric  Infrastructure  (NCI),  addressing  physical  and 
information  dimensions  of  NCW 

•  Note  that  NCW  almost  automatically  puts  every  weapon  system  in  a 
System  of  Systems  (SOS)  context 

-  Service-Oriented  Architecture  (SO A) 

•  SOA  is  an  emerging  architecture  style  that  may  be  used  to  implement 
Network  Centered  Infrastructure  (NCI).  Note  that  NCI  is  strongly 
promoted*  by  the  Office  of  the  Undersecretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (OUSD(AT&L)) 

*  Source:  [OUSD  2008],  also  see  “Net  Ready”  as  a  standard  Key  Performance 

Parameter  (KPP)  in  [DOD  2008] 
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Emerging  Software  Technologies  to  Watch  (cont.) 


•  Software  technology  breakthroughs  exploiting  advancements  in 
hardware  research  and  development 

-  Software  for  polymorphic  computing 

•  These  are  hardware  architectures  that  adapt  to  a  particular  objective 

-  Multi-threading 

•  True  concurrent  execution  of  threads  on  multiprocessors 

-  Software  designed  for  low  power  consumption 

•  There  are  many  possible  code  sequences  that  accomplish  the  same  task; 
the  objective  of  this  research  is  to  find  code  sequences  that  consume  the 
least  power  and  energy 

•  Software  process  technologies 

-  Model  Driven  Development  (MDD) 

-  Real-time  garbage  collection;  real-time  Java 

•  Enables  the  development  of  small  footprint,  high  assurance  software 

-  Incremental  Commitment  Process* 

-  Agile  Development 

•  For  details  on  this  new  process  see  [Pew  2007] 
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Algorithms 


•  Basic,  conventional  definition 

-  An  algorithm  is  a  sequence  of  finite  instructions.  It  is  formally  a  type  of 
effective  method  in  which  a  list  of  well-defined  instructions  will,  when  given 
an  initial  state,  proceed  through  a  well-defined  series  of  successive  states, 
eventually  terminating  in  an  end-state. 

•  Classification  of  algorithms  from  a  TRA  perspective* 

-  Domain-specific  algorithms 

•  Domain-specific  algorithms  implement  various  tasks  in  the  user’s 
domain 

-  Software  algorithms 

•  Software  algorithms  implement  various  tasks  in  the  software 
development  domain 

•  Implementation  variations  for  software  algorithms 

-  New  code 

-  Reuse  code 

-  Commercial  Off-the-Shelf  (COTS)  and  Government  Off-the-Shelf 
(GOTS)  software 

*Some  source  of  confusion  is  that  domain-specific  algorithms  might  be  implemented  in 
hardware,  firmware,  or  software  (See  filter  example  later.)  Also,  domain-specific 
algorithms  are  often  tested  with  software  tools,  but  that  does  not  make  them  software 
algorithms. 
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What  is  In-scope  for  a  Software  TRA? 


Domain-specific 

Aigorithm 


Not  Software 
Technoiogy 


Legend:  %  Red-colored  items  are  never  Software  CTE  candidates 
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Algorithm  Example:  Filters 


•  Definitions 

-  Filters  in  Signal  Processing 

•  A  filter  is  a  mathematical  algorithm  to  remove  part(s)  of  a  signal 

-  In  most  cases  the  goal  is  to  remove  interfering  noise  to  facilitate 
easier  processing  of  the  objective  signal 

-  Analog  filter 

•  Analog  filters  are  analog  electronic  circuit  implementations  of  filter 
algorithms,  operating  on  continuous-time,  analog  signals 

-  Digital  filter 

•  Digital  filters  are  digital  electronic  circuit  or  software  implementations  of 
filter  algorithms,  operating  on  sampled,  discrete-time,  digital  signals 

-  Application-specific  Integrated  Circuits  (ASICs) 

•  ASICs  are  customized  integrated  circuits  for  a  particular  application  rather 
than  intended  for  general-purpose  use 

-  Field  Programmable  Gate  Arrays  (FPGAs) 

•  An  FPGA  is  a  semiconductor  device  that  can  be  configured  by  the 
customer.  FPGAs  contain  programmable  logic  components  called  logic 
blocks,  and  a  hierarchy  of  reconfigurable  interconnects  that  allow  the 
blocks  to  be  "wired  together"  like  a  one-chip  programmable  breadboard. 
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Taxonomy  of  Filter  Implementations 


Hardware 


Software 


Domain-Specific  Aigorithms 
and  impiementations 


Domain-Specific  Aigorithms 
and  impiementations 

0AEROSmCE 
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Environmental  Considerations 


•  Environmental  categories  according  to  the  Deskbook: 

-  External  or  imposed  environment 

•  Related  to  the  operation  of  the  product,  may  be  either  natural  or  man¬ 
made 

-  Internal  or  designed  environment 

•  Always  man-made,  related  to  the  designed  product 

•  Further  environmental  dimensions 

-  Physical  environment  (for  software,  it  is  the  designed  environment) 

-  Logical  environment 

-  Data  environment 

-  Security  environment 

-  User  and  use  environment 

•  Recommendation: 

-  Base  software  environmental  analysis  on  the  software  architecture 
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What  Can  We  Learn  From  the  Software  Architecture?* 


•  Structural  Viewpoint 

-  Elements,  components  of  the  system 

-  Interactions  between  these  components  (“connectors”) 

-  Structural  organization  of  elements 

•  Behavioral  Viewpoint 

-  Dynamic  actions  of  and  within  the  system 

-  Actions  produced  by  the  system 

-  The  ordering  and  synchronization  of  these  actions 

-  The  behavior  of  system  components  and  their  interactions 

•  Physical  Interconnect  Viewpoint 

-  Physical  communication  interconnects  among  system  components 

-  Layering  among  system  components 

-  Feasibility  of  construction,  compliance  with  standards,  and  evolvability 


Note  that  all  this  information  is  relevant  to  technology  selection  and  evaluation 
but  even  an  elaborate  Work  Breakdown  Structure  (WBS)  would  not  show  it 


*  Discussion  is  based  on  the  prevailing  ISO/IEC  architecture  standard  [ISO/IEC  2007] 

0AEROSmCE 
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Selected,  High-Level  Viewpoints  for  TRA 


System  ^ 
Software 


Hardware 

Platform 


1/  / 


Application  Programming  Interfaces  (APIs) 

Operating 
System  & 
Libraries 

Middle¬ 

ware 

Services 

Infrastructure 

Communications 

Infrastructure 

SW 

Driver 

SW 

Driver 

Memory 

& 

Storage  Devices 

Processor 
(Computing  Node) 

Communications  HW 
(Bus  or  other 
Interconnections) 

HW 

IN 

HW 

OUT 

Structural  Viewpoint:  Software  Deployed  on  a  Single  Node 


Bus 


Tools 


Application  Programming  Interfaces  (APIs) 


Operating 
System  & 
Libraries 

Middle¬ 

Services 

Communications 

SW 

SW 

ware 

Infrastructure 

Infrastructure 

Driver 

Driver 

Memory 

Pror#»ssor 

Communications  HW 

HW 

HW 

& 

Storage  Devices 

(Computing  Node) 

(Bus  or  other 
Interconnections) 

IN 

OUT 

Physical  Interconnect  Viewpoint:  Software  Deployed  on  Multiple  Nodes 


*  Note  that  the  diagrams  are  intentionally  general  to  cover  any  kind  of  system,  including 
computers  on  airplanes  and  ships.  Space  customization  is  discussed  in  the  Case  Studies. 
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Dependency  of  Software  Environment  Layers 


System  ^ 
Software 


Hardware 

Platform 
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0 AEROSPACE 


CTE  Identification  Questions* 


2) 

3) 

4) 

5) 

6) 


Does  the  technology  have  a  significant  impact  on  an  operational 
requirement,  cost,  or  schedule? 

Does  this  technology  pose  a  major  development  or  demonstration 
risk? 

Is  the  technology  new  or  novel? 

Has  the  technology  been  modified  from  prior  successful  use? 

Has  the  software  technology  been  repackaged  such  that  a  new 
relevant  environment  is  applicable? 

Is  the  technology  expected  to  operate  in  an  environment  and/or 
achieve  a  performance  beyond  its  original  design  intention  or 
demonstrated  capability? 

A  technology  element  is  critical  if  the  answer  to  the  first 
question  and  to  any  of  the  remaining  questions  is  “yes” 


*  Source:  DOD  TRA  Deskbook,  pp  B-8 
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Disciplines  and  Knowledge  Involved  in  TRL  Determination 


# 

Basic 

TRL  DEFINITIONS 
from 

DOD  TRA  Deskbook* 

TRL 

GOALS 

Knowledge 
Involved  in 
Achieving 

HW  Objectives 

Knowledge 
Involved  in 
Achieving 

SW  Objectives 

Overarching 

Systems 

Engineering 

Responsibilities 

1 

Basic  principles  observed 
and  reported 

Demonstrate 

scientific 

feasibiiity 

Natural  Sciences 

Computer  Science 

N/A 

2 

Technology  concept 
and/or  application  formulated 

3 

Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept 

4 

Component  and/or  breadboard 
validation  in  a 
laboratory  environment 

Demonstrate 

engineering 

feasibiiity 

Hardware  Engineering 

Systems  Engineering 

Software  Engineering 

Systems  Engineering 

In-domain  integration 

Cross-domain  evaluation 

5 

Component  and/or  breadboard 
validation  in  a 
relevant  environment 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 

7 

System  prototype 
demonstration  in  an 
operational  environment 

Demonstrate 

operationai 

feasibiiity 

Hardware  Engineering 

Systems  Engineering 

Software  Engineering 

Systems  Engineering 

Cross-domain  integration 

8 

Actual  system  completed  and 
qualified  through  test  and 
demonstration 

9 

Actual  system  proven  through 
successful  mission  operations 

Demonstrate 

operations 

Mission  Domain 

Mission  Domain 

Mission  Domain 
Demonstration 

*  Note  that  the  basic  definitions  and  goals  for  software  and  hardware  are  the  same 
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Software  Technology  Readiness  Determination 


•  All  Software  TRLs  are  to  be  evaluated  on  the  same,  7  dimensions 

-  These  dimensions  are  recognized  drivers  oftechnoiogy  maturity 
demonstration.  The  objective  is  to  track  a  maturation  signature  via 
evoiution  in  these  dimensions 

-  A  TRL  is  deciared  to  be  achieved  if  objective  evidence  has  been  provided 
that  the  status  of  specific  conditions  supports  the  satisfaction  of  the  TRL  goai 
and  definition  as  stated 

•  Software  TRL  evaluation  dimensions 

-  Artifacts 

-  Structurai  Context 

-  Software  Environment 

-  Vaiidation  Environment  and  Methods 

-  Data  used  for  Vaiidation 

-  Configuration  Management 

-  Documentation 

See  evaluation  details  on  the  following  slides 
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Evaluating  Artifacts 


TRL 

Basic  TRL  DEFINITIONS 

Artifacts 

1 

Basic  principles  observed  and 
reported 

Research  articles,  peer-reviewed  white  papers,  point  papers,  early  conceptual 

models 

2 

Technology  concept  and/or 
application  formulated 

Analytic  studies,  papers  on  competing  technologies 

3 

Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept 

Analytical  and  simulation  models; 

Availability  of  appropriate  COTS/GOTS  or  reusabie  software  artifacts  is 

explored. 

4 

Component  and/or  breadboard 
validation  in  a  laboratory 
environment 

A  stand-alone  prototype  solving  a  partial-scale  problem;  Proposals  for  a 
nominal  software  architecture  and  for  a  simulation/stimuiation  work-up  plan; 
COTS/GOTS,  or  reusable  SW  if  applicable 

5 

Component  and/or  breadboard 
validation  in  a  relevant 
environment 

A  stand-alone  prototype  solving  a  partial-scale  problem;  Detailed  software 
architecture  and  final  simulation/stimulation  work-up  plan;  COTS/GOTS,  or 

reusable  SW  if  applicable 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 

Viable  prototype  providing  the  foundation  for  productization 

7 

System  prototype 
demonstration  in  an 
operational  environment 

Productized  component 

8 

Actual  system  completed  and 
qualified  through  test  and 
demonstration 

Productized  component 

9 

Actual  system  proven  through 
successful  mission 
operations 

Productized  component 

These  tables  are  structured  to  help  tracking  the  maturation  signatures 
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Evaluating  the  Structural  Context  and  SW  Environment 


TRL 

Basic  TRL  DEFINITIONS 

Structural 

Context 

SW 

Environment 

1 

Basic  principles  observed  and 
Reported 

n/a 

“Academic”,  experimental 

2 

Technology  concept  and/or 
application  formulated 

n/a 

“Academic”,  experimental 

3 

Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept 

n/a 

“Academic”,  experimental 

4 

Component  and/or  breadboard 
validation  in  a  laboratory 
environment 

Prototype  SW  Component 

“Academic”,  experimental 

5 

Component  and/or  breadboard 
validation  in  a  relevant 
environment 

Prototype  SW  Component 

Operational-like 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 

Prototype  WBS  Level  3 

Operational-like 

7 

System  prototype  demonstration 
in  an  operational  environment 

WBS  Level  2 

Actual  SW  Environment 

8 

Actual  system  completed  and 
qualified  through  test  and 
demonstration 

WBS  Level  1 

Actual  SW  Environment 

9 

Actual  system  proven  through 
successful  mission  operations 

WBS  Level  1 

Actual  SW  Environment 
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Evaluation  of  the  Validation  Environment  and  Data 


TRL 

Basic  TRL  DEFINITIONS 

Validation  Environment 
and  Methods 

Data  Used  for  Validation 

1 

Basic  principles  observed  and 
Reported 

Basic  research  using  analytical 
methods 

n/a 

2 

Technology  concept  and/or 
application  formulated 

Applied  research  using  analytical 
methods 

Synthetic  data  only 

3 

Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept 

Active  R&D  initiated  via  the  use  of 
models  and  simulation 

Partially  representative  data 

4 

Component  and/or  breadboard 
validation  in  a  laboratory 
environment 

Advanced  technology 
development  with  throwaway  or 
evolutionary  SW  prototypes 

Representative  data 

5 

Component  and/or  breadboard 
validation  in  a  relevant 
environment 

Advanced  technology 
development  with  throwaway  or 
evolutionary  SW  prototypes 

Representative  data 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 

Development  using  evolutionary 
SW  Prototype 

High-fidelity  data  representative  of 
relevant  environment 

7 

System  prototype 
demonstration  in  an  operational 
environment 

End-to-end  testing  of  Production 
SW  using  system  simulator 

High-fidelity  data  representative  of 
relevant  environment 

8 

Actual  system  completed  and 
qualified  through  test  and 
demonstration 

Testing  of  Production  SW  using 
OT&E 

Real  data 

9 

Actual  system  proven  through 
successful  mission 
operations 

Actual  Mission 

Real  data 
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Evaluation  of  CM  and  Documentation 


TRL 

Basic  TRL  DEFINITIONS 

Configuration 

Management 

Documentation 

1 

Basic  principles  observed  and 
reported 

n/a 

Appropriate  and  sufficient  to  demonstrate 
basic  principles 

2 

Technology  concept  and/or 
application  formulated 

n/a 

Appropriate  and  sufficient  to  demonstrate 
application  concept 

3 

Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept 

n/a 

Appropriate  and  sufficient  to  interpret 
analytical  or  experimental  data;  Full 
documentation  available  on  COTS/GOTS 
or  reusable  software  under  consideration. 

4 

Component  and/or  breadboard 
validation  in  a  laboratory 
environment 

Limited  scope;  appropriate  for 
experimental  environment 

Appropriate  and  sufficient  to  interpret 
experimental  results;  Full 
documentation  available  on  chosen 
COTS/GOTS  or  reusable  software 

5 

Component  and/or  breadboard 
validation  in  a  relevant 
environment 

Appropriate  for  operational-like, 
production  environment 

Appropriate  and  sufficient  to  interpret 
results;  Full  documentation  on  chosen 
COTS/GOTS  or  reusable  software 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 

Appropriate  for  operational-like, 
production  environment 

Appropriate  and  sufficient  to  validate 
relevant  environment  and  interpret 
demonstration  results 

7 

System  prototype  demonstration 
in  an  operational  environment 

Appropriate  for  the  actual 
environment 

Appropriate  and  sufficient  to  validate  test 
results 

8 

Actual  system  completed  and 
qualified  through  test  and 
demonstration 

Appropriate  for  the  actual 
environment 

Appropriate  and  sufficient  to  validate 
technology’s  performance  and  to  operate 
and  maintain  the  product. 

9 

Actual  system  proven  through 
successful  mission 
operations 

Consistent  with  the  actual 
environment 

Appropriate  and  sufficient  to  validate 
technology’s  performance  and  to  operate 
and  maintain  the  product. 
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0  AEROSPACE 

Assuring  Space  Mission  Success 


“In  Space  No  One  Can  Hear  You  Scream” 

- (Tagline  of  the  1979  Movie  “Alien”) 


Software  CTE  identification  for  a  Space  Vehicle 


Space 

System 


Space 

Segment 


Ground 

Launch 

User 

Segment 

Segment 

Segment 

•  Assumptions  and  Constraints 

-  The  example  space  vehicle  is  simplified  and  hypothetical 

-  Only  selected,  “mission-neutrar  satellite  functions  will  be  discussed 
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Architectural  Documentation-related  Definitions* 


•  Unified  Modeling  Language  (UML®) 

-  UML  is  a  language  for  visualizing,  specifying,  constructing,  and  documenting 
the  artifacts  of  a  software-intensive  system 

•  Context 

-  Context  includes  all  those  things  that  on  the  outside  interact  with  the  system 

•  Actor 

-  An  Actor  represents  a  role  that  a  human,  a  hardware  device,  or  another 
system  plays  with  the  objective  system 

-  The  Actor  symbol  in  UML  is  as  follows: 

A 

•  Deployment  Diagram 

-  A  diagram  showing  the  nodes  that  form  the  system’s  hardware  topology  on 
which  the  system  executes 


*  Source:  [Booch  1999] 

®  UML  is  Registered  in  the  U.S.  Patent  and  Trademark  Office  by  The  OMG,  Inc. 
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Partial*  Space  Vehicle  Flight  Software  Context  Diagram 


Navigation 

Navigation  Data 


Control  System 
Operator 

Operator  Commands 
and  Telemetry  Data 

Timing  Reference 
Control 

Maintenance  & 
Distribution  of  clock 


Power  Control 

Sensors/Controllers 


Crosslink 

Misc.  telemetry, 
timekeeping,  payload 
Messages 


Space  Vehicle 
Flight  Software 


/ 


/ 


Thermal  Control 

Sensors/Controllers 


On-board 

Recovery 

Recovery  and 
Redundancy 


Attitude  Control 

Sensors/Actuators 


Communication 

Ground  to  Space 
Vehicle 


Communication 

Space  Vehicle  to 
Ground 
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*  Only  selected,  “mission-neutrar  satellite  functions  are  shown 
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Sample  Worksheet  for  Software  CTE  Identification 


FUNCTIONAL  AREA 

COMMENTS 

ARE  THERE  ANY 
SOFTWARE  CTE 
CANDIDATES?* 

Attitude  Control 

•  Reading  attitude  sensors 

•  Control  attitude  actuators 

Most  likely  no 

Communication  (External) 

•  Space  Vehicle  to  Ground 

•  Ground  to  Space  Vehicle 

Maybe 

(Depends  on  mission) 

Communication  (Internal) 

•  Interfacing  with  other  payloads 

Most  likely  no 

Crosslink  Processing 

•  Miscellaneous  telemetry, 
timekeeping,  and  payload 
messages  across  satellites 

Most  likely  no 

Data  Base  Management 

Most  likely  no 

Payload  Bus  Controller 

Most  likely  no 

Diagnostics  and  Maintenance 

•  Recovery  management 

•  Redundancy  processing 

Most  likely  no 

Electrical  Power  Subsystem  Control 

•  Sensors 

•  Controllers 

Most  likely  no 

Keeping  Time  (Clock) 

•  Maintaining  time  information 

Most  likely  no 

Navigation/Management  of  the  Space  Vehicle  (SV) 

•  Reading  telemetry  data 

•  Commanding  the  SV 

Most  likely  no 

Thermal  Control 

•  Sensors 

•  Controllers 

Most  likely  no 

etc. 

*  Note  that  in  case  of  any  potential  candidates  the  6  CTE  identification  questions  would 
have  to  be  answered  for  final  classification 
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UML  Deployment  Diagram  for  the  Space  Vehicle 


•  Payload  Bus  Controller  is  usually  a  COTS  solution 

•  Other  payloads  could  be  miscellaneous  communications,  imagery,  infra¬ 
red  radar,  etc.  processing,  depending  on  the  actual  mission 
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TRA-Oriented,  Customized  Deployment  Diagram* 


Tools 


Software  Applications  for 

SV  Subsystem  Control 

-  Command  processing 

-  Electrical  Power  Subsystem  processing 

-  Navigation  and  Control  data  processing 
-Telemetry  data  collection  and  processing 
-Thermal  data  processing 

-  etc. 


No 

Human 

Interface 


Real-time  Operating 
System 

Communications 

Infrastructure 

SW  Drivers 

SW  Drivers 

Memory 

Processor 

Communications 

HW 

(Bus) 

HW  IN 

-  Attitude  sensors 

-  Thermal  sensors 

-  Power  sensors 

HW  OUT 

-  Attitude  actuators 

-  Thermal  control 

-  Power  control 

Bus 


Tools 


Software  Applications  for 

SV  Communications 

■  Crosslink  processing 

■  Database  management 

■  Payload  to  Payload  communication 

■  SV  to  Ground  Communication 

■  Timekeeping  processing 

■  etc. 


No 

Human 

Interface 


^ - - - 

Real-time  Operating 
System 

Communications 

Infrastructure 

SW  Drivers 

SW  Drivers 

Memory 

Processor 

Communications 

HW 

(Bus) 

HWIN 

-  Crosslink 
-  Clock 

HWOUT 

-  Crosslink 

•  Conclusions: 

”  Apparently  no  need  for  “extreme’’  new  or  novel  solutions 
—  For  sake  of  simplicity  we  assume  that  no  reuse  or  COTS  are  planned 

—  We  still  need  to  examine  dependency  and  tools  issues 

*  Note  that  distributing  functionality  across  processors  is  an  architecting  step.  The  deployed 
functionality  on  the  processor  does  not  always  neatly  match  the  subsystem  name. 
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Detour:  Dealing  with  Radiation  in  Space  Hardware 


•  Why  is  electromagnetic  interference  a  problem? 

-  In  presence  of  ionizing  radiation  a  single  charged  particle  can  knock 
thousands  of  electrons  loose 

•  Major  radiation  damage  sources  in  space 

-  Solar  particle  events 

-  Van  Allen  radiation  belts 

-  Cosmic  rays,  etc. 

•  Selected  effects  of  radiation  on  electronics 

-  Rearrangement  of  the  atoms,  creating  lasting  damage 

-  Transient  ionization  effect  or  latchup 

•  During  a  single-event  latchup,  heavy  ions  or  high-energy  protons  are 
passing  through  the  inner-transistors  of  the  circuit,  causing  a  “short” 

•  Radiation  hardening  (“rad-hard”) 

-  It  is  a  method  of  designing  and  testing  electronic  components  to  make  them 
resistant  to  electromagnetic  radiation  damage 
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Radiation  Hardening 


•  Selected  radiation  hardening  approaches* 

-  Shielding 

•  Involves  connplete  shielding  of  assemblies  and  subsystems 

-  Radiation-Hardening-by-Process  (RHBP) 

•  This  is  the  “classic”  approach  to  rad-hard 

•  “Process”  refers  to  the  semiconductor  fabrication  process 

-  Chips  made  on  special,  insulating  substrates  like  silicon  oxide  or 
sapphire 

-  Choice  of  substrate  with  wide  band  gap 

-  Radiation-Hardening-by-Design  (RHBD) 

•  The  radiation  mitigation  techniques  are  implemented  in  layout  or  in  the 
chip  architecture  and  not  in  the  fabrication  process 

-  Augmenting  reliability  techniques 

•  Using  redundant  elements  on  circuit-  and/or  system  level 

•  Applying  a  watchdog  timer  to  perform  a  hard  reset  if  needed 

*  Reference:  [Barnaby  2004] 
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Software  Technology  Readiness  Implications 


•  Shielding  is  the  only  solution  without  software  implications 

-  However,  it  is  bulky  and  heavy 

•  Remember,  excess  weight  carries  a  special  penalty  in  space  designs 

•  In  case  of  all  other  solutions  exercise  caution 

-  Many  vendors  developed  rad-hard  processors  that  are  based  on  (and 
claimed  to  be  equivaient  with)  various,  commercial  processors,  such  as  the 
IBM  PowerPC.  However,  they  have  to  be  evaluated  on  a  case-by-case  basis 

•  For  real-time  applications  the  development  environment,  including  the 
used  compiler,  has  to  be  validated  on  the  same  rad-hard  platform 

•  With  respect  to  the  objective  system,  the  main  concern  is  the  preservation 
of  real-time  characteristics  when  the  software,  at  the  last  phase  of  the 
development  process,  is  moved  from  a  commercial,  developer  platform  to 
a  rad-hard  platform  (“Operational-like”  Software  Environment) 

•  Conclusion 

-  The  understanding  of  hardware  technoiogy  options  and  their  maturity  is 
essential  for  a  successful  software  TRA 
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The  Dependency  of  Software  TRLs  on  Hardware  TRLs 


PL 

(Programming 

Language) 

DE 


(Development 

Environment) 


Board) 


TRL(PL)  <  TRL(DE)  <  TRL(OS)  <  TRL(HW) 


The  equation  above  saves  the  effort  of  fully  understanding  the  details  of  the  particular  layers  © 

0 AEROSPACE 
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0  AEROSPACE 

Assuring  Space  Mission  Success 


“...And  there  is  nothing  new  under  the  sun. 

Is  there  anything  of  which  it  may  be  said,  ’See,  this  /snew’? 
It  has  already  been  in  ancient  times  before  us.” 

~~~  Ecclesiastes  1 :9-1 0 


SOA  as  a  CTE  in  a  Ground  System 


•  SOA  as  a  CTE? 

-  Google  produced  40  million  (!)  hits  in  0.2  sec  for  “SOA”.  Even  if  we  discount 
hits  on  the  Society  of  Actuaries  and  such,  it  is  very  impressive.  Wouldn't  it 
prove  that  it  is  a  mature  technology? 

-  No.  Using  SOA  is  a  risky  proposition  and  extreme  caution  is  needed.  SOA 
belongs  to  the  category  where  concepts  and  new  code,  reuse,  and  COTS 
dimensions  ali  might  have  to  be  evaluated. 
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Detour:  What  is  a  Service-Oriented  Architecture  (SOA)? 


•  Architecture* 

-  “Architecture  is  the  fundamental  organization  of  a  system  embodied  in  its 
components,  their  relationships  to  each  other  and  to  the  environment,  and 
the  principles  guiding  its  design  and  evolution.  ” 

•  Service-Oriented  Architecture** 

-  “A  Service-Oriented  Architecture  takes  advantage  of  networking  capabilities 
to  integrate  applications  in  a  way  that  is  independent  of  architecture, 
programming  language,  development  platform  and  vendor.  Through  a  set  of 
standard  interfaces,  services  are  made  available  to  any  consumer  willing  to 
follow  the  rules  for  interface  and  consumption.  ” 

•  Selected,  generic  SOA  services 

-  Messaging,  mediation/translation  between  data  structures  and  protocols. 
Data  Base  Management  System  (DBMS,)  high-speed  networking, 
collaboration.  Information  Assurance/Security,  etc. 

•  Question  to  ponder 

-  What  do  you  think  the  benefits  of  using  such  an  architectural  style  are? 

*  Source:  [ISO/IEC  2007] 

**  Source:  [Minkiewicz  2007] 
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The  Road  to  SOA  for  Space 


•  SOA  is  a  promising  approach  to  implement  Operational  Responsive 
Space  (ORS)  and  Joint  Warfighting  Space  (JWS) 

•  Operational  Responsive  Space 

-  ORS  is  characterized  by  an  incrementai  approach  from  prototyping  to 
production,  on  the  basis  of  highiy  moduiarized  capabiiities 

-  According  to  the  eariy  ORS  ideas,  the  key  to  achieving  these  objectives  is 
space  system  bus*  standardization 

•  Note  that  the  term  “bus”  in  ORS  equally  relates  to  all  segments  of  a  space 
system,  not  only  to  the  space  vehicle. 

•  Joint  Warfighting  Space  [Schuler  2005] 

-  The  JWS  initiative  seeks  to  make  space  an  organic  part  of  joint  task  forces  in 
theater 

-  ORS  is  an  enabier  of  JWS 

•  Question  to  ponder 

-  What  do  you  think  the  connection  is  between  ORS  and  the  eariier  mentioned 
NOW  doctrine? 

*  ORS  was  originally  introduced  by  the  now  defunct  Office  of  Force  Transformation 

(OFT)  DOD  entity.  Here  we  only  refer  to  the  generic  aspects  of  the  earlier  OFT  proposal. 
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Selected  SOA  Services  for  a  Ground  System 


•  Sample  Ground  functionality  that  could  be  implemented  via  services* 

-  Command  Processing 

•  Commands  to  space  vehicles 

•  Commands  to  antennae  systems 

-  Orbital  Data  Processing 

-  Critical  alarms 

-  Mission  Planning 

-  Real-time  Telemetry  Processing 

-  Processing/providing  data  on  external  interfaces 

•  Other  ground  stations 

•  External  clients  of  ground  station  services 

-  Situational  awareness 

-  Etc. 

•  Of  course  a  SOA  Framework  would  be  needed  as  well 

-  E.g.,  to  implement  the  registry  that  facilitates  the  seamless  integration, 
upgrade,  discovery  and  invocation  of  services 

•  Caveat:  SOA  does  not  always  make  sense  for  implementing  all  Ground  System  functionality 
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SOA  Components  in  a  Ground  System 


Human 

Interface 


Application  Programming  Interfaces  (APIs) 


Communications 

Infrastructure 


Services  Infrastructure 


Middle 


(SOA  Framework) 


ware 


Operating 

SystenrSr 

Libraries 


Memory 

& 

Storage  Devices 


Processor 
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Tools 


Communications  HW 
(Bus) 


Software  Applications 


Legend:  Q  Potential  SOA  Component 

Note  that  this  GS  example  does  not  have  HW  input/output  elements  other  than  the  bus 
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CTE  Identification  Questions  Revisited 


2) 

3) 

4) 

5) 

6) 


Does  the  technology  have  a  significant  impact  on  an  operational 
requirement,  cost,  or  schedule? 

Does  this  technology  pose  a  major  development  or  demonstration 
risk? 

Is  the  technology  new  or  novel? 

Has  the  technology  been  modified  from  prior  successful  use? 

Has  the  software  technology  been  repackaged  such  that  a  new 
relevant  environment  is  applicable? 

Is  the  technology  expected  to  operate  in  an  environment  and/or 
achieve  a  performance  beyond  its  original  design  intention  or 
demonstrated  capability? 

A  technology  element  is  critical  if  the  answer  to  the  first 
question  and  to  any  of  the  remaining  questions  is  “yes” 
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CTE  Identification  Question  1 


(1)  Does  the  technology  have  a  significant  impact  on  an  operational 
requirement,  cost,  or  schedule? 

•  Regarding  operational  requirements  the  answer  is  definitely  YES, 
due  to  the  Net-Ready  KPP  and  the  earlier  mentioned  OUSD  SOA 
directive  implications 

•  Also,  regarding  Cost  and  Schedule,  service-orientation  provides 
numerous  enablers  for  improvement 

-  The  framework  and  core  building  blocks  can  be  acquired  as  COTS 

-  Services  allow  substantial  reuse 

-  A  properly  designed  SOA  framework  enables  the  automatic  discovery  of 
fine-grained  software  services,  negotiates  their  acquisition,  and 
composes,  binds,  executes,  and  unbinds  them 

-  The  use  of  SOA  speeds-up  the  incremental  delivery  of  new  capabilities 
in  the  system  evolution  context 

-  Through  loose  coupling  and  tight  interface  standards,  consumers  of 
services  need  only  know  how  to  interact  with  a  service,  and  there  is  no 

_ need  to  understand  deeper  details 
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CTE  Identification  Questions  2-4 


(2)  Does  this  technology  pose  a  major  development  or  demonstration 
risk? 

•  YES.  SOA  is  very  pervasive,  the  associated  components  show  up 
in  many  different  parts  of  the  overall  system 

(3)  Is  the  technology  new  or  novel? 

•  NO.  Do  you  remember  Distributed  Object  Architecture  (DOA,) 
Common  Object  Model  (COM,)  Object  Request  Broker  (ORB,) 
Common  Object  Requerst  Broker  Architectrure  (CORBA,)  etc.? 

(4)  Has  the  technology  been  modified  from  prior  successful  use? 

•  It  depends.  However,  most  likely  the  answer  is  NO,  because  the 
SOA  COTS  elements  would  not  be  modified 
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CTE  Identification  Questions  5-6 


(5)  Has  the  technology  been  repackaged  such  that  a  new  relevant 
environment  is  applicable? 

•  Maybe.  “Repackaging”  is  a  broad  term  and  involves 
considerations  for  both  the  hardware  and  systems  software 
platforms  of  the  SOA  services.  This  question  is  particularly  critical 
when  services  are  independently  adopted  for  the  objective  system 

(6)  Is  the  technology  expected  to  operate  in  an  environment  and/or 
achieve  a  performance  beyond  its  original  design  intention  or 
demonstrated  capability? 

•  YES.  One  of  the  main  concerns  with  SOA  implementations  is  the 
overhead  associated  with  service  invocation/execution  and  its 
impact  on  performance  and  Quality  of  Service  (QoS).  It  is  very 
unlikely  that  the  structure  of  the  objective  system  would  be  identical 
to  a  prior  system  already  in  use;  hence  these  factors  should  be 
verified  in  advance 
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Evaluation  of  Answers 


1 )  Does  the  technology  have  a  significant  impact  on  an  operational 
requirement,  cost,  or  schedule?  YES 

2)  Does  this  technology  pose  a  major  development  or  demonstration 
risk?  YES 

3)  Is  the  technology  new  or  novel?  NO 

4)  Has  the  technology  been  modified  from  prior  successful  use?  NO 

5)  Has  the  software  technology  been  repackaged  such  that  a  new 
relevant  environment  is  applicable?  Maybe 

6)  Is  the  technology  expected  to  operate  in  an  environment  and/or 
achieve  a  performance  beyond  its  original  design  intention  or 
demonstrated  capability?  YES 

Conclusion:  Due  to  YES  answers  to  questions  1 ,  2,  and  6,  SOA  is  a  CTE 


Reminder:  A  technology  element  is  critical  if  the  answer  to  the  first  question  and  to  any  of  the 
remaining  questions  is  “yes” 
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More  SOA  Specifics 


•  The  6  questions  need  to  be  carefully  applied  and  special  attention  to  be 
paid  to  the  following  specifics: 

-  Platform/Relevant  Environment  compatibility  for  COTS  SOA  elements 

•  Note  that  different  services  might  come  from  different  sources 

-  Inter-component  Compatibility  of  acquired  COTS  SOA  elements 

•  See  reason  given  at  the  prior  issue 

-  Execution  performance  considerations  and  scaling  of  services 

•  SOA  involves  a  lot  of  overhead;  as  it  was  mentioned  earlier,  the  impact 
needs  to  be  carefully  assessed  in  advance 

-  SOA  interface  adequacy  for  new  services 

•  Service  uniformity  is  achieved  via  standard  interfaces;  however,  beyond 
the  core  functionality,  the  inherent  throughput,  data  capacity,  error 
tolerance,  testability,  etc.  might  not  be  enough  for  the  new  service. 

-  Feasibility  and  effort  needed  to  interface  reused  and  legacy  software 

•  Due  to  the  fact  that  most  likely  only  selected  functionality  would  be 
provided  as  a  “service” 
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COTS/Reuse  Evaluation  Revisited* 


•  The  application  of  COTS/Reused  Software  is  a  very  attractive  but  at  the 
same  time  very  risky  approach  to  software  development 


-  Consequently,  we  need  clarity  on  what  COTS/Reuse  Attributes  are  In-scope 
for  a  TRA  and  what  inquiries  belong  to  the  “routine”,  programmatic  risk 
management  area 


In-scope  for  TRA 

-  Accuracy,  correctness 

-  Availability 

-  Robustness 

-  Security 

-  Product  performance 

-  Testability 

-  Version  compatibility 

-  Inter-component  Compatibility 

-  Functionality 


In-scope  for  Trades  and  Risk  Reduction 

-  Documentation  quality 

-  Ease  of  Use 

-  Flexibility 

-  Installation/Upgrade  Ease 

-  Portability 

-  Price 

-  Vendor  support  and  maturity 

-  Training 

-  Vendor  concessions 


*  Note  that  the  assessment  of  evaluation,  glue  code  writing,  and  integration  cost(effort  and 
schedule)  is  also  out  of  scope  for  a  TRA  while  it  is  a  very  critical  planning  activity 
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Challenging  Topics 

(What  they  Don’t  Teach  at  the  Defense  Acquisition  University...) 


“Technological  change  is  like  an 
axe  in  the  hands  of  a  pathological 

criminal.” 

- Albert  Einstein 


Challenging”  is  a  Euphemism  for  Controversial.. 


Sources  of  controversy 

-  Incorrect  or  ambiguous  government  guidance 

-  Lack  of  consensus  amongst  the  practitioners 

-  Mismanaged  expectations 

Issues  to  be  discussed 


-  The  consequences  of  rolling-up  TRLs 

•  The  need  to  roll-up  TRLs  many  times  manifests  itself  as  a  call  for 
providing  a  single  system  maturity  index 

-  A  sub-issue  that  needs  to  be  addressed  is  the  attempt  for  algebraic 
manipulation  of  an  ordinal  measurement  scale 

-  Putting  too  much  faith  into  the  predictive  ability  of  TRLs 

•  The  underlying  issue  is  to  understand  that  technology  development 
should  be  viewed  as  an  innovation  process  and  as  such  it  is  highly 
unpredictable 

-  TRA  vs.  Risk  Management 

•  We  will  look  at  a  particular  example,  programming  language  selection  for 
high  assurance  software,  where  they  are  in  conflict 

-  The  confusion  between  Technology  Maturity  and  Software  Maturity 

•  We  will  look  at  the  reasons,  i.e.,  confusing  definitions  and  lack  of 
understanding  of  the  systems  and  software  engineering  life  cycle 
processes 


Rolling-up  TRLs 


•  Rolled-up  TRLs  are  supposed  to  provide  one  single  maturity  number  for 
a  system  or  a  SOS 

-  The  idea  equally  relates  to  a  single  system  with  multiple  CTEs  or  a  SOS  with 
dominant  CTEs  (or  rolled-up  TRLs.  )  assigned  to  the  participating  systems 

•  Question:  What  would  be  your  recommendation  for  a  roll-up  rule? 

a)  Use  the  lowest  TRL 

TRLsys  =  min  (TRL^,  TRL2,  TRLJ 

b)  Compute  the  arithmetic  average  of  TRLs 


TRL 


SYS  ~ 


TRL^i  -{TRL2  “1“  ---  ~\~TRL^ 
n 


c)  I  have  another  method  to  do  the  roll-up 

d)  I  don’t  recommend  rolling-up  TRLs 
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Rolling-up  TRLs:  My  Answers  in  the  order  of  preference 


•  (d)  Don’t  roll-up  TRLs 

-  No  real  need  for  roll-up 

•  If  CTE  selection  has  been  correctly  carried  out  then  we  should  not  have 
too  many  technology  elements  to  deal  with 

-  It  is  beneficial  to  keep  the  maturity  evolution  of  CTEs  separately  visible 

•  However,  in  case  of  dependency  they  need  to  be  collectively  re-evaluated 

•  (a)  Use  the  lowest  TRL 

-  This  approach  is  based  on  well-known  reliability  principles 

•  If  all  selected  CTEs  are  indeed  critical*  then  the  weakest  link  should  be 
used  to  describe  the  maturity  of  the  overall  system  or  SOS 

•  (b)  Arithmetic  average  of  TRLs 

-  Not  acceptable.  One  mustn’t  do  arithmetic  on  ordinai  scaies  under  any 
circumstances 

•  (c)  Do  you  have  any  suggestions? 

•  Remember,  the  6  CTE  identification  questions  should  narrow  down  the  candidate  CTE  list 
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TRL  as  an  Ordinal  Scale 


TRL  9 

TRL  8 

TRL  7 

TRL  6 

TRL  5 


TRL  4 

TRL  3 


TRL  2 


•  Unlike  in  “classic”  risk  prioritization  schemes,  a  single,  1-9 
integer  is  assigned  to  every  maturity  level 

-  For  any  CTE  the  impact  of  not  moving-up  from  any  levei  to 
the  next  within  the  program’s  cost  and  scheduie  constraints  is 
the  loss  of  the  total  program 

•  In  conventional  risk  management 

-  Risks  are  characterized  with  their  likelihood  and  their  impact 

-  Risk  burn-down  plans  can  be  put  in  place  to  reduce  or 
eliminate  risks 

•  What  do  the  TRLs  represent? 

-  TRLs  represent  a  certain  level  of  growing  confidence  that  the 
technology  will  perform  as  intended.  However,  we  cannot 
quantify  this  confidence  beyond  the  “warm  feeling”  what  an 
ordinal  scale  could  provide 

•  We  don’t  know  the  likelihood  of  success  at  any  levels 

•  We  cannot  easily  estimate  the  costs  either 

-  Details  to  follow  on  the  next  few  slides 


61 


0 AEROSPACE 


A  Creative  Problem  Solving  Profile  for  Innovation* 


Action 


Problem 

finding 


Accept, 

Fact 

sell  idea 

finding 

Plan 

Problem 

definition 

Evaluate 
and  select 


•  Despite  the  diagram’s  neat 
symmetry,  in  reality  the  steps  in 
the  slices  have  different 

-  Durations 

-  Activity  types 

-  Leveis  of  complexity 

-  Levels  of  difficulty 

•  In  fact,  according  to  Basadur, 
the  profile  not  only  changes  by 
problem  type,  but  it  is  also 
different  individual  by  individual 

•  Phase  4  is  the  most  volatile 


*  Source:  [Basadur  2001] 
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Technology  Development  as  an  Innovation  Process 


TRL9 

TRL8 

TRL7 

TRL6 

TRL5 


TRL4 


TRk3 


TRt2 


TRL 


E.g.,  moving  from  TRL  1  ^  TRL  2  requires  the 

application  of  the  8-step  problem  solving  process  to 

•  Studying  and  documenting  competing  technologies 

•  Creating  an  experimental  environment 

•  Determining  what  research  methods  should  be  used 

•  Determining  and  acquiring  data  needed  for  validation 

•  Finally,  documenting  how  the  concept  will  be  applied 


Note  the  “red  slices”  for  every  level  where  the  need  for  individual  creativity  is  particularly  critical 
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...  but  Wouldn’t  Phase  4  Volatility  Gradually  Decrease? 


•  One  might  intuitively  think  that  when  a  CTE  matures,  the  uncertainty 
associated  with  the  needed  innovation  gradually  decreases  and 
estimation  predictability  increases 

•  However 

-  This  idea  is  based  on  the  ideaiistic  assumption  that  aii  CTEs  are  totaiiy 
independent  in  reaiity,  mainiy  at  and  above  TRL  6,  we  have  to  worry  about 
potentiai  interference  from  other  CTEs  or  in  some  cases  even  from  other, 
non-criticai  system  eiements 

•  This  potential  for  interference  is  gradually  increasing  as  the  expanding 
structural  context  dimension  of  the  rating  scheme  shows 

•  The  need  for  creative,  innovative  thinking  is  decreasing 

-  in  fact,  CTEs  often  have  to  be  recursiveiy  re-evaiuated  or  even 
fundamentaiiy  changed  as  their  maturity  evaiuation  and  the  overaii  system 
impiementation  progress 
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And  Finally,  an  Expert’s  Voice  ... 


•  Richard  Feynman*  was  once  asked  how  many  process  steps  he  used 
during  problem  solving.  After  some  thinking,  he  said  that  his  “secret” 
process  formula  constituted  the  following  three  steps: 

1)  Write  down  the  problem 

2)  Think  real  hard 

3)  Write  down  the  solution 


*  Richard  Feynman  (1918-1988);  professor  and  researcher;  one  of  the 
most  famed  physicist  of  the  world.  Besides  his  work  on  the  atomic  bomb  at 
the  Manhattan  Project,  he  became  also  known  as  a  key  member  of  the 
panel  that  investigated  the  Space  Shuttle  Challenger  disaster.  The 
included  picture  is  a  copy  of  his  old  government  ID  from  Los  Alamos 
(Photograph  reprinted  courtesy  of  the  United  States  Government.) 
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TRA  and  Risk  Management  Revisited 


•  We  made  an  effort  to  neatly  separate  TRA  and  Risk  Management 

-  However,  there  are  various  situations  for  interpiay,  synergy,  and  even 
confiict 

•  Example  for  synergistic  relationship 

-  There  are  numerous  items  that  are  not  intentionaiiy  in-scope  for  a  TRA,  but 
the  iRT  might  get  insight  into  during  attendance  of  the  contractors’  design 
reviews,  studying  research  documents,  etc. 

-  It  is  the  IRT  members’  duty  to  go  beyond  the  core  reporting  on  critical 
technologies  and  provide  risk  warnings  to  program  management  on  any 
technology-related  risk  that  has  been  discovered  during  a  TRA 

•  Example  for  interplay 

-  In  case  of  COTS,  TRA  is  viewed  as  an  early  screening  activity  to  narrow 
down  the  list  of  COTS  options  for  trade  studies  and  risk  reduction  activities 

•  Example  for  conflict 

-  Selection  of  a  programming  language/compiler  for  flight  software  (details  to 
follow.) 
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Ada  or  C++  For  Developing  High  Assurance  Software? 


Level  of  Protection  Within  the  Languages* 


Criteria 

C++ 

Ada  95 

Wild  Jumps 

Some 

Few 

Overwrites 

Some 

Few 

Semantics 

Some 

Good 

Model  of  Maths 

None 

Good 

Operational  Arithmetic 

None 

Good 

Data  Typing 

Some 

Good 

Exception  Handling 

Some 

Good 

Safe  Subsets 

None 

Good 

Exhaustion  of  Memory 

Some 

Rare 

Separate  Compilation 

Some 

Good 

*  The  issue  is  that  there  is  a  higher  probability  of  violating  good  programming  practices  with 
C++  than  with  a  more  declarative  language  like  Ada  95.  For  criteria  details  please  see  next  slide. 
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Criteria  for  Ada/C++  Comparison 


Wild  Jumps 

-  The  program  cannot  Jump  to  an  arbitrary  memory  location 

Overwrites 

-  The  program  cannot  overwrite  an  arbitrary  memory  location 

Semantics 

-  The  compiler  supports  static  code  analysis 

“Model  of  Maths” 

-  A  rigorous  model  for  floating-point  and  integer  arithmetic 

Operational  Arithmetic 

-  Run-time  checking  to  ensure  adherence  to  “model  of  maths” 

Data  Typing 

-  Typing  model  is  strong  enough  for  static  checking  and  run-time  checking 

Exception  Handling 

-  Mechanisms  to  facilitate  recovery  from  run-time  faults  are  provided 

Safe  Subsets 

-  A  language  subset  exists  that  increases  the  safety  of  the  language 

Exhaustion  of  Memory 

-  A  mechanism  exists  to  ensure  that  the  program  does  not  run  out  of  memory 

Separate  compilation 

-  Type  checking  is  provided  across  all  parts  of  the  development  environment 
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Analysis:  Technology  vs.  Programmatic  Risks  with  Ada 


•  Even  though  it  seems  that  Ada  is  superior  for  high-assurance, 
embedded  system  development,  there  are  several  reasons  for  the 
program  manager  to  vote  against  Ada  (Factors  not  supposed  to  be 
considered  during  a  TRA): 

-  Ada  programming  skills  are  becoming  scarce 

•  Universities  shun  Ada;  they  believe  that  there  is  no  market  for  Ada  skills 

•  There  is  a  disdain  toward  Ada  amongst  programmers  that  is  partially  a 
backlash  due  to  the  earlier  DOD  Ada  mandate 

•  Ada  programmers  have  difficulty  in  getting  jobs  outside  of  DoD 

-  The  available  selection  of  development  environments  and  tools  for  Ada  Is 
becoming  narrower  and  more  expensive 

•  We  need  to  balance  between  what  the  technology  buys  vs.  other  factors 
that  could  be  more  relevant  in  assessing  programmatic  risks 

-  The  programmatic  risks  associated  with  adopting  Ada  may  be  higher  than 
with  adopting  C++  even  though  Ada  clearly  poses  a  lower  technology  risk 
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Software  Technology  Maturity  is  Software  Maturity 


•  Software  Technology  Maturity  definition  revisited 

-  A  measure  of  degree  to  which  proposed  concepts,  processes,  methods, 
algorithms,  or  tools,  whose  primary  purpose  is  the  development,  operation, 
understanding,  and  maintenance  of  software,  meet  program  objectives 

•  Software  Maturity  (Software  Product  Maturity  or  Technical  Maturity) 
characteristics  are  drastically  different 

-  The  form  of  software  evolves  rather  than  being  replaced  by  artifacts  of 
different  forms  (note  the  breadboards  ->  brassboards  ->  manufactured  items 
metamorphosis  in  case  of  hardware) 

•  In  nnost  cases  -  unlike  hardware  -  software  is  developed  iteratively 

-  All  software  artifacts  continuously  and  concurrently  evolve 

-  Software  integration  is  different  from  hardware  integration 

•  Software  integration  is  a  continuous  activity  from  the  beginning  of  the 
evolution  of  the  objective  system 

-  One  does  not  “make”  every  component  first  and  then  “integrate” 

-  Software  Product  Maturity  is  not  an  ever-increasing  measure 

•  Software  can  become  “senile” 

-  At  some  point  in  its  life  cycle,  the  number  of  changes  and  bug-fixes 
can  overwhelm  the  architecture  and  the  software  starts  degrading 
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Software  Technology  Maturity  is  Software  Maturity  (cent.) 


•  The  distinction  is  not  simply  about  mincing  words 

-  Technology  maturity  has  an  important  gating  role  leading  up  to  MS  B, 
while  planning  and  monitoring  software  product  evolution  are  essential 
after  MS  B. 

•  However,  the  current  TRL  definitions  conflate  these  topics 

-  TRLs  1-4  focus  on  traditional  technology  innovation-type  activities 

-  TRLs  5-7  focus  on  the  fidelity  of  test/validation  environments  in  relation 
to  integration  and  Verification  and  Vaiidation  (V&V)  type  activities 

-  TRLs  8-9  focus  on  productization  and  deployment  activities 

•  Is  this  a  problem  for  hardware  elements  of  a  program? 

-  Maybe,  but  this  tutoriai  is  about  software  © 

•  Is  this  a  problem  for  software  elements  of  a  program? 

-  Yes.  TRLs  1-4  reflect  indeed  software  technologies,  but  the  shift  to  V&V 
between  TRL  4  and  5  is  meant  to  be  applied  to  a  software  product 
rather  than  a  software  technology 
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Concluding  Thoughts 


•  Technology  Readiness  Assessment,  a  critical  tool  in  Defense 
Acquisition,  helps  us  to  prevent  the  incorporation  of  immature 
technologies  during  system  design 

•  Modern  weapon  systems  are  software-intensive  systems; 
consequently,  the  early  evaluation  of  software  technologies  is  essential 

•  Software  Technology  Readiness  Assessments  are  carried  out  by 
software  specialists.  However 

-  An  understanding  of  the  maturity  of  aii  rotated,  hardware  technotogies  is 
necessary 

-  During  Technoiogy  Readiness  Assessments  a  carefut  coordination 
amongst  the  technicat  area  speciatists  is  atso  a  must 
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I  am  Sorry  if  You  are  Still  Confused  ... 


I  am  sure  I  have  failed  to  anticipate  or  answer  all  of  your  questions.  I  am  sorry  if 
it  seems  that  the  answers  I  provided  only  served  to  raise  a  new  set  of  questions. 
In  some  ways  you  might  feel  that  you  (and  I...)  are  still  as  confused  as  ever,  but  I 
do  believe  that  now  we  are  confused  on  a  much  higher  level  and  about  more 
important  things. .  .Thank  you  again  for  attending  the  tutorial! 
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Acronyms 


ACQ 

Acquisition 

API 

Application  Programming  Interface 

APO 

Acquisition  Program  Office 

ASIC 

Application  Specific  Integration  Circuit 

AT&L 

Acquisition,  Technology,  &  Logistics 

CASE 

Computer-Assisted  Software  Engineering 

CM 

Configuration  Management 

CMMI 

Capability  Maturity  Model  Integration 

COTS 

Commercial  Off-the-Shelf 

CTE 

Critical  Technology  Element 

DAPA 

Defense  Acquisition  Performance  Assessment 

DBMS 

Data  Base  Management  System 

DDR&E 

Director  of  Defense  Research  &  Engineering 

DE 

Development  Environment 

DOD 

Department  of  Defense 

DAB 

Defense  Acquisition  Board 

FPGA 

Field-Programmable  Gate  Array 

GOTS 

Government  Off-the-Shelf 

GUI 

Graphical  User  Interface 

HW 

Hardware 

lEC 

International  Electrotechnical  Commission 

IRT 

Independent  Review  Team 

ISO 

International  Standards  Organization 

JWS 

Joint  War-fighting  Space 

KDP 

Key  Decision  Point 

MDD 

Model  Driven  Development 

MIL 

Military 

MOIE 

Mission-Oriented  Investigation  and  Experimentation 
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NCI 

Network  Centric  Infrastructure 

NCO 

Network  Centric  Operations 

NCW 

Network  Centric  Warfare 

NSS 

National  Security  Space 

NSSAP 

National  Security  Space  Acquisition  Policy 

OFT 

Office  of  Force  Transformation 

OMG 

Object  Management  Group 

ORS 

Operational  Responsive  Space 

OS 

Operating  System 

OT&E 

Operational  Test  &  Evaluation 

OUSD 

Office  of  the  Under  Secretary  of  Defense 

PL 

Programming  Language 

PSP 

Personal  Software  Process 

R&D 

Research  &  Development 

RHBD 

Radiation  Hardening  by  Design 

RHBP 

Radiation  Hardening  by  Process 

RPC 

Remote  Procedure  Call 

SAF/AQ 

Secretary  of  the  Air  Force/Acquisition 

SEAM 

Systems  Engineering  Assessment  Model 

SMC 

Space  and  Missile  Systems  Center 

SOA 

Service  Oriented  Architecture 

STD 

Standard 

SW 

Software 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 

UML 

Unified  Modeling  Language 

use 

United  States  Code 

V&V 

Verification  &  Validation 
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Assuring  Space  Mission  Success 


Relevant  Environment  for  Space 


•  Space  Environment  Attributes 

-  Pressure  (vacuum) 

-  Temperature  (deep  space) 

-  Gravity  (microgravity  on  orbit) 

-  Radiation 

-  Eiectromagnetic  Spectrum  (star  tight,  sunlight,  x-ray,  etc.) 

•  Launch  Environment  Attributes 

-  Pressure  (e.g.,  changing  pressure,  venting) 

-  Temperature  (e.g.,  payload  fairing  heating) 

-  Gravity  (acceleration,)  Vibration,  Structural  Loads 

•  Designed  Environment  Attributes 

-  Physical  internal  environment  (e.g.,  self  generated  heating) 

-  Software  interaction  with  environment  (processor  throughput,  memory 
capacity,  bus  bandwidth,  etc.) 

-  Hardware  Interfaces  (e.g.,  thermal  shorts) 

-  Behavior  of  model  or  prototype  in  interfacing  environment  (dynamic 
behavior) 

•  Operational  Environment  Attributes 

-  Lifetime,  Duty  Cycles,  Complex  Dynamic  Behavior 
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TRL  1-3  Evaluation  Dimensions 


TRL 

Basic  TRL 
DEFINITIONS 

Artifacts 

Structural 

Context 

sw 

Environment 

Validation 
Environment 
and  Methods 

Data  Used  for 
Validation 

Configuration 

Management 

Documentation 

1 

Basic  principles 
observed  and 
reported 

Research  articles, 
peer-reviewed  white 
papers,  point  papers, 
early  conceptual 
models 

n/a 

“Academic”, 

experimental 

Basic 

research 

using 

analytical 

Methods 

n/a 

n/a 

Appropriate 
and  sufficient 
to 

demonstrate 

basic 

principles 

2 

Technology 

concept  and/or 
appifcation 
formulated 

Analytic  studies, 
papers  on  competing 

technologies 

n/a 

“Academic”, 

experimental 

Applied 

research 

using 

analytical 

Methods 

Synthetic  data 
only 

n/a 

Appropriate 
and  sufficient 

to 

demonstrate 

application 

concept 

3 

Analytical  and 
experimental 
critical  function 
and/or 

characteristic 
proof  of  concept. 

Analytical  and 
simulation  models; 
Availability  of 
appropriate 
COTS/GOTS  or 
reusable  software 
artifacts  is  explored. 

n/a 

“Academic”, 

experimental 

Active  R&D 
initiated  via 
the  use  of 
models  and 
simulation 

Partially 

representative 

data 

n/a 

Appropriate 
and  sufficient 
to  interpret 
analytical  or 
experimental 
data;  Full 
documentation 
is  available  on 
COTS/GOTS 
or  reusable 
software 
under 

consideration. 

These  tables  help  to  evaluate  all  factors  associated  with  a  TRL 
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TRL  4-6  Evaluation  Dimensions 


TRL 

Basic  TRL 
DEFINITIONS 

Artifacts 

Structural 

Context 

SW 

Environment 

Validation 
Environment 
and  Methods 

Data  Used  for 
Validation 

Configuration 

Management 

Documentation 

4 

Component  and/or 
breadboard 
validation  in  a 
laboratory 
environment 

A  stand-alone 
prototype  solving  a 
partial-scale  problem; 
Proposals  for  a 
nominal  software 
architecture  and  for  a 
simulation/stimulation 
work-up  plan; 
COTS/GOTS,  or 
reusable  SW  if 
applicable 

Prototype 

SW 

Component 

“Academic”, 

experimental 

Advanced 

technology 

development 

with 

throwaway 

or 

evolutionary 

SW 

prototypes 

Representative 

data 

Limited 

scope; 

appropriate 

for 

experimental 

environment 

Appropriate 
and  sufficient 
to  interpret 
experimental 
results; 

Full 

documentation 
on  chosen 
COTS/GOTS 
or  reusable 
software 

5 

Component  and/or 
breadboard 
validation  in  a 
relevant 
environment 

A  stand-alone 
prototype  solving  a 
partial-scale  problem; 
Detailed  software 
architecture  and  final 
simulation/stimulation 
work-up  plan; 
COTS/GOTS,  or 
reusable  SW  if 
applicable 

Prototype 

SW 

Component 

Operational- 

like 

Advanced 

technology 

development 

with 

throwaway 

or 

evolutionary 

SW 

prototypes 

Representative 

data 

Appropriate 

for 

operational- 

like, 

production 

environment 

Appropriate 
and  sufficient 
to  interpret 
results;  Full 
documentation 
on  chosen 
COTS/GOTS 
or  reusable 
software 

6 

System/subsystem 
model  or  prototype 
demonstration  in  a 
relevant 
environment 

Viable  prototype 
providing  the 
foundation  for 
productization 

Prototype 
WBS  Level 

3 

Operational- 

like 

Development 

using 

evolutionary 

SW 

Prototype 

High-fidelity 

data 

representative 
of  relevant 
environment 

Appropriate 

for 

operational- 

like, 

production 

environment 

Appropriate 
and  sufficient 
to  validate 
relevant 
environment 
and  interpret 
demonstration 
results 

The  red  border  emphasizes  the  special  position  TRL  6  occupies  in  the  Technology  Readiness 
Assessment  Process 
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TRL  7-8  Evaluation  Dimensions 


TRL 

Basic  TRL 
DEFINITIONS 

Artifacts 

Structural 

Context 

sw 

Environment 

Validation 
Environment 
and  Methods 

Data  Used  for 
Validation 

Configuration 

Management 

Documentation 

7 

System  prototype 
demonstration  in 
an  operational 
environment 

Productized 

component 

WBS  Level 

2 

Actual  SW 
Environment 

End-to-end 
testing  of 
Production 

SW  using 

system 

simulator 

High-fidelity 

data 

representative 
of  relevant 
environment 

Appropriate 
for  the  actual 
environment 

Appropriate 
and  sufficient 
to  validate  test 
results 

8 

Actual  system 
completed  and 
qualified  through 
test  and 
demonstration 

Productized 

component 

WBS  Level 

1 

Actual  SW 
Environment 

Testing  of 
Production 

SW  using 
OT&E 

Real  data 

Appropriate 
for  the  actual 
environment 

Appropriate 
and  sufficient 
to  validate 
technology’s 
performance 
and  operate 
and  maintain 
the  product. 
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TRL  9  Evaluation  Dimensions 


TRL 

Basic  TRL 
DEFINITIONS 

Artifacts 

Structurai 

Context 

sw 

Environment 

Validation 
Environment 
and  Methods 

Data  Used  for 
Validation 

Configuration 

Management 

Documentation 

9 

Actual  system 
proven  through 
successful  mission 
operations 

Productized 

component 

WBS  Level 

1 

Actual  SW 
Environment 

Actual 

Mission 

Real  data 

Consistent 
with  the 
actual 

environment 

Appropriate 
and  sufficient 
to  validate 
technology’s 
performance 
and  operate 
and  maintain 
the  product. 

81 


0 AEROSPACE 


Use  of  Trademarks,  Service  Marks  and  Trade  Names 


Use  of  any  trademarks  in  this  material  is  not  intended  in  any 
way  to  infringe  on  the  rights  of  the  trademark  holder.  All 
trademarks,  service  marks,  and  trade  names  are  the  property 

of  their  respective  owners. 
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